🕷️️ Job Radar • SCRAPING

freelancer.com 2026-04-26 🟡

🔹 Automated Weekly Product Data Scraping
👤 Client: 🇦🇪 Dubai, United Arab Emirates Member since 2026-04-26
💰 Price: $130 Average bid
🚩 Problem: Need to reliably gather product information from multiple websites weekly.
📦 Existing: Not specified

Specifications:

[Target] Specific URLs of target websites for scraping
[Method] Use Python with Scrapy and Selenium for dynamic content handling
[UI/UX] Not applicable
[Stack] Python, Scrapy, Selenium, BeautifulSoup, Pandas, Openpyxl, Boto3
[Security] Secure credentials storage (environment variables or encrypted files)
[Format] CSV, JSON, Google Sheets, S3

Workflow:

1. Define target URLs and data points needed: SKU, product title, current price, availability, description, image URLs.
2. Set up Scrapy project with custom spiders for each URL, using Selenium for dynamic content handling.
3. Implement data cleaning and deduplication logic in Python script.
4. Schedule the scraping process to run weekly using cron or Windows Task Scheduler.
5. Export scraped data to CSV and JSON files.
6. Push data to Google Sheets and S3 bucket.